Cluster data replication

ABSTRACT

A cluster data replication system includes a plurality of network elements and controllers. The controllers form a cluster that is able to elect one of the controllers as a master controller with the others being follower controllers. The elected controller updates, responsive to being elected the master controller, state information in a system database of the elected controller to indicate that the elected one of the controllers is the master controller. The master controller includes one or more objects that are enabled in reaction to the state information, and which coordinate replication of changes to the data, system database, and state information from the master to the follower controllers. Each follower controller includes one or more objects able to, in reaction to the state information, disable initiation of the replication of changes to the data, system database and state information by the one or more objects in each follower controller.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. application Ser. No.16/806,666, filed Mar. 2, 2020, entitled “Cluster File Replication,”which is a continuation of U.S. application Ser. No. 15/296,851, filedOct. 18, 2016, entitled “Cluster File Replication.” The entire contentsof each of these applications is incorporated herein by reference forall purposes.

BACKGROUND

Data coherency across multiple controllers in a cluster is vital forreliable operation. If one controller loses a database or a file, orceases operation, the database or file could be rebuilt, or thecontroller could be rebooted, but the system state and various pieces ofdata could change meanwhile. In network controllers, mappings of virtualnetworks should be persisted, but can experience loss of data coherencyunder the above conditions. In high-availability controller clusters, aswitchover from master to follower could incur delays if data has lostcoherency and has to be downloaded again or reconstructed. There is alsoa problem of determining which controller should be a source for datareplication, when the master can change under various conditions. Ablanket policy of frequent data copying, if not well-coordinated, couldresult in chaotic data. Also, frequent data copying consumes system andprocessor bandwidth. File deletion, under data replication, can beproblematic. Therefore, there is a need in the art for a solution whichovercomes the drawbacks described above.

SUMMARY

In some embodiments, a cluster file replication system is provided. Eachcontroller of a plurality of controllers of the system is configured toaccess a filesystem having a plurality of files including a systemdatabase of a controller having state information of the plurality ofcontrollers. Each controller is further configured to have one or moreservice agents. The one or more service agents of each controller isconfigured to respond to one of the plurality of controllers becoming amaster controller of the cluster The one or more objects on each of thefollower controllers is supportive of the follower controllers receivingthe changes but disabled from initiating the replication.

In some embodiments, a tangible, non-transitory, computer-readable mediahaving instructions thereupon which, when executed by one or moreprocessors in a cluster of controllers, cause the one or more processorsto perform a method. The method includes writing, by one controller inthe cluster, state information to a system database of the onecontroller to record that the one controller is a master controller. Themethod includes establishing, by one or more service agents of eachcontroller in the cluster, one or more objects in the controller thatare reactive to the state information. The method includes coordinating,by the one or more objects established in the master controller andenabled in reaction to the state information, replication of change tofiles, the system database or the state information from the mastercontroller to follower controllers, such that the one or more objects oneach of the follower controllers supports receiving the replication ofchange but is disabled, in reaction to the state information, frominitiating the replication of change.

In some embodiments, a method for cluster file replication, performed bycontroller members of a cluster is provided. The method includesupdating, by a controller that is a member of the cluster, stateinformation in a system database of the controller to indicate thecontroller is a master controller, and establishing, by one or moreservice agents of each controller, one or more objects in the controllerthat react to the state information. The method includes coordinating,by the one or more objects in the master controller enabled in reactionto the state information, replication of changes to files, the systemdatabase and the state information from the master controller tofollower controllers. The method includes disabling, by the one or moreobjects in each of the follower controllers in reaction to the stateinformation, initiation by the follower controllers of such replication,while being supportive of the follower controllers receiving suchchanges.

Other aspects and advantages of the embodiments will become apparentfrom the following detailed description taken in conjunction with theaccompanying drawings which illustrate, by way of example, theprinciples of the described embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The described embodiments and the advantages thereof may best beunderstood by reference to the following description taken inconjunction with the accompanying drawings. These drawings in no waylimit any changes in form and detail that may be made to the describedembodiments by one skilled in the art without departing from the spiritand scope of the described embodiments.

FIG. 1 is a system diagram of a cluster of network controllers, withnetwork elements, showing use of a high-availability controller.

FIG. 2 is a block diagram of a high-availability controller that has afilesystem, one or more applications and one or more service agents thatinstantiate objects from a file replication library to coordinatereplication of system changes from a master controller to followercontrollers in systems such as shown in FIGS. 1 and 3-5 .

FIG. 3 is a system diagram depicting a replication object in a mastercontroller coordinating with the filesystem to replicate changes offiles, system database and/or state information from the mastercontroller to follower controllers in a cluster.

FIG. 4A shows an embodiment of a responsive replication object suitablefor use as the replication object in FIG. 3 .

FIG. 4B shows an embodiment of a periodic replication object suitablefor use as the replication object in FIG. 3 .

FIG. 5 depicts a controller joining the cluster, and propagation ofupdated state information as a result.

FIG. 6 is a flow diagram of a method for cluster file replication, whichcan be performed by controller members of a cluster, includingembodiments shown in FIGS. 1-5 and variations thereof.

FIG. 7 is an illustration showing an exemplary computing device whichmay implement the embodiments described herein.

DETAILED DESCRIPTION

Cluster file replication, as presented herein in various embodiments,synchronizes files between cluster members. Files are synchronized fromthe cluster leader, a master controller, to followers, i.e., followercontrollers. Service agents, for example a license manager, use thebelow-described mechanisms to make sure that any state stored on thefilesystem is persisted across the cluster and available on the newleader or master if there is a leader or master switchover. Files to bekept in sync are registered with the cluster by a service agent.Whenever a file changes on the leader, i.e., the master controller, thechanges are propagated to the followers, i.e., the follower controllers,without further intervention from the service agent that set up thereplication mechanism. Likewise, system state, which the master canchange, is persisted and available on the followers as well. Embodimentsare described below, first with reference to a cluster of networkcontrollers, and then genericized to controllers in a cluster thatpersists data and system state across the cluster, which could beapplicable to various systems that use multiple controllers. Controllersdescribed herein solve the technical problem of recognizing who is themaster and then replicating data transparently.

FIG. 1 is a system diagram of a cluster of network controllers 102, withnetwork elements 110, showing use of a high-availability controller 102.One of the controllers 102 is a master 116, the others are followers118. Each controller has a network controller database 104, which iskept coherent across the controllers 102 and across the network elements110, each of which has a network element database 112. Each networkelement 110 could have one or more servers 120 coupled to it, or not,depending upon the function(s) of that network element 110. Applications108 in the master 116 network controller 102 are typicallynetwork-related and are used for setting up, configuring and managing anetwork or multiple virtual networks formed by the network elements 110.These applications 108 can make changes to various files and/or to thenetwork controller database 104. The network controllers 102 haveelected one of the network controllers 102 as the master 116 or masternetwork controller 102, and the others are followers 118 or followernetwork controllers 102. Embodiments of the controller 102, as furtherdescribed below, keep the network controller database 104 and otherfiles, as well as the system state, coherent across the controllers 102.If the master network controller 102 drops out temporarily orpermanently, e.g. due to any of a variety of failures, the remainingcontrollers 102 elect a new master 116, which takes up operation usingthe coherent state and coherent data. The cluster of network controllers102 cooperating with network elements 110 as shown in FIG. 1 is but oneexample of a suitable environment for the controllers 102 described withreference to FIGS. 2-6 , and further systems for these controllers 102are readily envisioned.

FIG. 2 is a block diagram of a high-availability controller 102 that hasa filesystem 202, one or more applications 108 and one or more serviceagents 106 that instantiate objects 216 from a file replication library214 to coordinate replication of system changes from a master 116controller 102 to follower 118 controllers 102 in systems such as shownin FIGS. 1 and 3-5 . The processor 218 could be one or more physicalprocessors in a physical controller 102, or could be threaded or timesliced from one or more processors 218 in a virtualized systemimplemented with physical computing components. Each controller 102,whether a master 116 or a follower 118, has or has access to files 208including a system database 210, which are accessed using a directorytree 206 through the kernel 204 of the filesystem 202. The systemdatabase 210 has various kinds of state information 212, includingindications of which controllers are members of the cluster, and foreach such controller, whether that controller 102 is a master 116 or afollower 118. This and other state information 212 and other systeminformation can be represented in various formats as readily devised. Insome embodiments, each service agent 106 has a file replication library214, from which the service agent can instantiate one or more objects216, according to any of various addressing schemes or spaces as readilydevised. In further embodiments, there is a single file replicationlibrary 214 on each controller 102, from which each service agent 106can instantiate one or more objects 216. Interactions of agents 106,objects 216 and kernels 204 in master 116 and follower 118 controllers102, for cluster file replication in a controller-based system, arefurther described below.

Various components in the controller 102 can be implemented in softwareexecuting on the processor 218 and/or as structures in memory in thecontroller 102. Further components can be implemented in firmware orhardware. In virtualized systems, the controller 102 can be implementedwith physical computing components. Software code, in some embodiments,for object(s) 216 instantiated by one service agent 106 could be in theaddress space of the service agent 106, or in another agent 106, orelsewhere in memory.

FIG. 3 is a system diagram depicting a replication object 310 in amaster 116 controller 102 coordinating with the filesystem 202 toreplicate changes 302 of files 208, system database 210 and/or stateinformation 212 from the 116 master controller 102 to follower 118controllers 102 in a cluster. The replication object 310 is instantiatedby a service agent 106, from a file replication library 214. Furtherreplication objects 310 could be instantiated by the same service agent106, or by other service agents 106. The service agent 106 thatinstantiated the replication object 310 registers one or more files ordirectories of interest 312, and the corresponding replication object310, with the kernel 204 of the filesystem 202. This action informs thekernel 204 to watch for changes to the one or more files or directoriesof interest 312.

The master controller 116 can make various changes 302 that affect thesystem state of the controller 102. Examples include writing a file(e.g., this could include writing an entire new file or writing aportion of a file), deleting a file, writing state information 212 tothe system database 212 (which is a type of file write), etc. A newmaster 116, upon being elected by the controllers 102, updates the stateinformation 212 to indicate that it has become the master 116. A master116 controller 102, upon being informed or otherwise discovering that acontroller 102 has joined the cluster as either a new controller 102 ora recovered controller 102 that had previously failed, updates the stateinformation 212 to so indicate. The master 112 can also write to thestate information 212 to indicate that a controller 102 is no longer amember of the cluster (e.g., that controller 102 has failed), or toindicate that a controller 102 that was formerly a master 112 is now afollower 118 (e.g., when the new master 112 takes over). Applications108 executing on the master 116 controller 102 can also write, delete orupdate files. Service agents 106 executing on the master 116 controller102 can write, delete or update files.

When the kernel 204 of the filesystem 202 of a master 116 controller 102detects, perceives or is informed of a change 302 in the filesystem 202(e.g., a file write, file move or file deletion), if this matches theregistering of the file(s) or directory/directories of interest 312, thekernel 204 sends the appropriate notification 304 to the appropriatereplication object 310 on the master 116 controller 102. Thatreplication object 310 sends a replicate request 306 (also called asynchronize request) to the kernel 204 on the master 116 controller 102.Upon receiving a replicate request 306, the kernel 204 on the master 116controller 102 sends change information 308, symbolized in the drawingby the Greek symbol “delta”, to each of the follower 118 controllers102. The kernel 204 of the filesystem 202 of the master 116 controller102 cooperates with the kernels 204 in the filesystems 202 in each ofthe followers 118 to deliver the change information 308, whichsynchronizes the file(s) or directory/directories of interest 312 fromthe master 116 controller 102 to the follower 118 controllers 102.

In the followers 118, the replication objects 310 are deactivated frominitiating replication, but are supportive of the follower 118controller 102 receiving the change information 308 (e.g., thereplication objects 310 do not interfere). Behavior in each of thecontrollers 102, for the replication objects 310 and service agents 106,is reactive to the state information 212. And, using the abovemechanisms, the state information 212 is synchronized across thecontrollers 102. So, replication object(s) 310 in the master 116controller 102 are enabled to initiate replication in reaction to thestate information 212 indicating that they are on the master 116controller 102, and are responsive to notification 304. Replicationobject(s) 310 in each of the follower 118 controllers 102 are disabledfrom initiating replication, in reaction to the state information 212indicating they are on follower 118 controllers 102, and areunresponsive to any notifications 304. By linking the behavior of thereplication objects 310 to the state information 212, which is coherentacross the master 116 and follower(s) 102, the flow of replication ofchange 302 only from master 116 to follower(s) 118, and not vice versa,is assured. This flow direction changes when a new master 116 iselected, and the new master propagates the state information 212accordingly, while the system continues to have coherence of data andstate under the new master 116.

Various utilities and commands in various operating systems could beused to implement aspects of the interactions among the components of acontroller-based cluster. For example, in Linux, inotify and rsync areused in some embodiments. As a Linux kernel subsystem, inotify (meaningMode notify) notices changes to a file system and reports the changes toapplications. To use inotify, the application registers with thefilesystem 202 kernel 204 the file(s) or directory/directories for whichnotification is desired, and the kernel 204 sends a notification to theapplication whenever there is a change to the file or files or directoryor directories. Depending on specific syntax for commands or utilitiesin a specified operating system, the notification request could be acombined command for all of the files or directories of interest, or asingle command for each file or directory. In some embodiments,replication objects 310 are the applications that get an inotify orother notification, which triggers the replication. This is preferred,so that the replication objects 310 can operate independently of theservice agents that instantiated them, and service agent 106intervention is not required for coherent file replication in thecluster. However, in further embodiments, service agents 106 are theapplications that get an inotify or other notification, which triggersthe replication. This latter approach supports coherent file replicationin the cluster, with ongoing service agent 106 involvement.

As a Linux utility, rsync is both a file synchronization and filetransfer program, which can be used to synchronize files and/ordirectories between different systems. The source system connects to thedestination system, and the two systems determine what parts of a fileneed to be transferred. An rsync request, made on a command line,specifies a source including a file or files or directory or directoriesin the source, and a destination file or files or directory to which tocopy. In some embodiments, the replication is performed using rsync.Other operating systems and other commands or utilities for replicationare readily applied in further embodiments.

One mechanism for ensuring there is no corruption during the replicationis to copy a file to a destination under a different, temporaryfilename, then once the file is completely copied to the destination,the temporary file is renamed thus moving the temporary file to thetarget under the desired filename. The renaming operation is an atomicrename of the file to the actual destination file, and is an existingimplementation or mechanism in some operating systems. With thismechanism, if the master 116 crashes during the middle of replication,the file might otherwise have been corrupted in the follower 118, butthis only corrupts the temporary file instead, and either the recoveringmaster 116 or a new master 116 can then take over and recover the systemcorrectly.

As part of replication, if the file is in the filesystem 202 of themaster 116 controller 102, the rsync or other replicate request 306synchronizes from the master 116 to the follower(s) 118. If a file hasbeen deleted on the master 116, there is a file delete on thefollower(s) 118. One way to service both of these possibilities is touse an inotify and treat this as a dirty flag, then check the filesystem202 to see if the file actually exists, and determine actionsaccordingly.

In some embodiments, an entire directory in which a file of interestresides is being monitored, so there are updates for all of the files inthat directory. Filenames of interest are tracked, and any updates offiles not of interest are discarded. With this specificity applied toreplication, only the files of interest are synchronized across thesystem, and system and processor bandwidth are not spent on replicatingfiles that are not of interest. This improves operating efficiency ofthe system.

In some embodiments, the system database 210 is in charge of sendingupdates to the service agents 106 whenever the state changes. Forexample, a service agent 106 on the master 116 controller 102 couldwrite to the system database 210 to change the state information 212,which triggers an update, and the system database 210 knows which otherservice agents 106 are listening for states and will send updates tothose service agents 106.

A system based on the controllers 102 has fault tolerance in that acontroller 102 can go off-line, then return, and the system updates thestate information 212 in that controller 102. The controller 102 reactsto the coherent state information 212 as described above, and thus doesnot operate out of synchrony with the other controllers 102, even thoughthat returning controller 102 has not directly experienced systemoperation or system state changes while the controller 102 wasnon-functioning.

FIG. 4A shows an embodiment of a responsive replication object 402suitable for use as the replication object 310 in FIG. 3 . In thisversion, each time the responsive replication object 402 receives aninotify 404 (or other notification 304), the responsive replicationobject 402 sends out an rsync 406 or other replicate request 306.Filesystem 202 synchronization from master 116 to follower(s) 118 thusoccurs with every change 302 of interest (i.e., change affecting theregistered file(s) or directory/directories of interest 312) detected bythe kernel 204.

However, in some systems this may result in too frequent updates andexcessive load on the network just to replicate files. This also may notscale well to larger systems. As a solution to these problems, theupdates that occur during a specified time interval could get collapsedinto one periodic update, as described below.

FIG. 4B shows an embodiment of a periodic replication object 408suitable for use as the replication object 310 in FIG. 3 . A timer 410,which could be specific to the periodic replication object 408, or couldbe based on intervals on a system timer, etc., establishes a timeinterval, which could be fixed, or programmable, or variable. During thetime interval, if one or more inotify 404 or other notification(s) 304arrive, the periodic replication object 108 is primed and issues anrsync 406 or other replicate request 306 upon completion or expirationof the time interval. If there are no notifications 304 during the timeinterval, there is no replicate request 306 at the end of the timeinterval. Example time intervals or rates could be once per second, onceper ten seconds, etc. In a further embodiment, the notifications 304could be counted, and the replicate request 306 sent upon reach of aspecified count, or time interval, whichever comes first. Furthervariations are readily devised in keeping with the teachings herein.

FIG. 5 depicts a controller 102 joining the cluster, and propagation ofupdated state information 212 as a result. In this scenario, the new orrecovered controller 102 joins the cluster and communicates itspresence, e.g., by sending a message such as an announcement, a greetingor an inquiry to other controllers 102, or by responding to anothercontroller 102, or by issuing a heartbeat, etc. An agent 106 on themaster 116 controller 102 detects that the new or recovered controller102 has joined the cluster, e.g., by receiving the message from the newor recovered controller 102, and sends an update 502 for the new orrecovered controller 102 to the state information 212. A change 302 inthe state information 212 is detected, as illustrated in FIG. 3 .Because of the change 302, the change information 308 is propagated fromthe master 116 to the followers 118, which now includes the follower 118controller 102 already present in the cluster and the new or recoveredcontroller 102. All controllers 102 then have consistent, coherent stateinformation 212 showing which controller 102 is the master of 116, andwhich controller(s) 102 are the followers 118. The scenario depicted inFIG. 5 (and FIG. 3 ) can also be used to illustrate what happens when anew master 116 is elected among the controllers 102, and, as part of theswitchover, a similar process of that master 116 updating the stateinformation 212 and propagating the change information 308 to thefollowers 118 occurs.

FIG. 6 is a flow diagram of a method for cluster file replication, whichcan be performed by controller members of a cluster, includingembodiments shown in FIGS. 1-5 and variations thereof. Steps or actionsin the method can be performed by a processor, such as a processor in orof the controller.

In an action 602, state information is updated to indicate a mastercontroller. For example, a controller, upon being elected by controllersin a cluster, could write to state information in a system database inthat controller to indicate that it is the master controller, and othercontrollers that are members of the cluster are follower controllers.

In an action 604, objects are set up that react to state information, inthe controllers. For example, in each controller in the cluster, one ormore service agents could set up one or more replication objects thatare reactive to state information.

In an action 606, one or more objects in the master controller areenabled. These could be the replication object or objects, and they areenabled in reaction to the state information indicating that they are onthe master controller.

In an action 608, one or more objects in each of the followercontrollers are disabled. These could be replication objects, and theyare disabled in reaction to the state information indicating they are onfollower controllers.

In an action 610, replication of changes to files, system database,state information etc. is coordinated by one or more objects in themaster controller. The replication of changes proceeds from the masterto the followers. For example, a replication object in the mastercontroller, reactive to the state information indicating it is on themaster controller, could receive a notification from the kernel of achange to a file or directory of interest which the master controller(e.g., an agent, or the replication object) has registered with thekernel, and respond by issuing a replication request to the kernel. Thekernel of the master controller could then replicate the changes, bysending change information from the master controller to the followercontrollers. In this manner, files, directories and state informationare replicated and kept coherent across all of the controllers.

It should be appreciated that the methods described herein may beperformed with a digital processing system, such as a conventional,general-purpose computer system. Special purpose computers, which aredesigned or programmed to perform only one function may be used in thealternative. FIG. 7 is an illustration showing an exemplary computingdevice which may implement the embodiments described herein. Thecomputing device of FIG. 7 may be used to perform embodiments of thefunctionality for cluster file replication in accordance with someembodiments. The computing device includes a central processing unit(CPU) 701, which is coupled through a bus 705 to a memory 703, and massstorage device 707. Mass storage device 707 represents a persistent datastorage device such as a floppy disc drive or a fixed disc drive, whichmay be local or remote in some embodiments. The mass storage device 707could implement a backup storage, in some embodiments. Memory 703 mayinclude read only memory, random access memory, etc. Applicationsresident on the computing device may be stored on or accessed via acomputer readable medium such as memory 703 or mass storage device 707in some embodiments. Applications may also be in the form of modulatedelectronic signals modulated accessed via a network modem or othernetwork interface of the computing device. It should be appreciated thatCPU 701 may be embodied in a general-purpose processor, a specialpurpose processor, or a specially programmed logic device in someembodiments.

Display 711 is in communication with CPU 701, memory 703, and massstorage device 707, through bus 705. Display 711 is configured todisplay any visualization tools or reports associated with the systemdescribed herein. Input/output device 709 is coupled to bus 705 in orderto communicate information in command selections to CPU 701. It shouldbe appreciated that data to and from external devices may becommunicated through the input/output device 709. CPU 701 can be definedto execute the functionality described herein to enable thefunctionality described with reference to FIGS. 1-6 . The code embodyingthis functionality may be stored within memory 703 or mass storagedevice 707 for execution by a processor such as CPU 701 in someembodiments. The operating system on the computing device may be MSDOS™, MS-WINDOWS™, OS/2™, UNIX™, LINUX™, or other known operatingsystems. It should be appreciated that the embodiments described hereinmay also be integrated with a virtualized computing system implementedwith physical computing resources.

Detailed illustrative embodiments are disclosed herein. However,specific functional details disclosed herein are merely representativefor purposes of describing embodiments. Embodiments may, however, beembodied in many alternate forms and should not be construed as limitedto only the embodiments set forth herein.

It should be understood that although the terms first, second, etc. maybe used herein to describe various steps or calculations, these steps orcalculations should not be limited by these terms. These terms are onlyused to distinguish one step or calculation from another. For example, afirst calculation could be termed a second calculation, and, similarly,a second step could be termed a first step, without departing from thescope of this disclosure. As used herein, the term “and/or” and the “/”symbol includes any and all combinations of one or more of theassociated listed items.

As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”,“comprising”, “includes”, and/or “including”, when used herein, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. Therefore, the terminology usedherein is for the purpose of describing particular embodiments only andis not intended to be limiting.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

With the above embodiments in mind, it should be understood that theembodiments might employ various computer-implemented operationsinvolving data stored in computer systems. These operations are thoserequiring physical manipulation of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated. Further, the manipulationsperformed are often referred to in terms, such as producing,identifying, determining, or comparing. Any of the operations describedherein that form part of the embodiments are useful machine operations.The embodiments also relate to a device or an apparatus for performingthese operations. The apparatus can be specially constructed for therequired purpose, or the apparatus can be a general-purpose computerselectively activated or configured by a computer program stored in thecomputer. In particular, various general-purpose machines can be usedwith computer programs written in accordance with the teachings herein,or it may be more convenient to construct a more specialized apparatusto perform the required operations.

A module, an application, a layer, an agent or other method-operableentity could be implemented as hardware, firmware, or a processorexecuting software, or combinations thereof. It should be appreciatedthat, where a software-based embodiment is disclosed herein, thesoftware can be embodied in a physical machine such as a controller. Forexample, a controller could include a first module and a second module.A controller could be configured to perform various actions, e.g., of amethod, an application, a layer or an agent.

The embodiments can also be embodied as computer readable code on atangible non-transitory computer readable medium. The computer readablemedium is any data storage device that can store data, which can bethereafter read by a computer system. Examples of the computer readablemedium include hard drives, network attached storage (NAS), read-onlymemory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes,and other optical and non-optical data storage devices. The computerreadable medium can also be distributed over a network coupled computersystem so that the computer readable code is stored and executed in adistributed fashion. Embodiments described herein may be practiced withvarious computer system configurations including hand-held devices,tablets, microprocessor systems, microprocessor-based or programmableconsumer electronics, minicomputers, mainframe computers and the like.The embodiments can also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a wire-based or wireless network.

Although the method operations were described in a specific order, itshould be understood that other operations may be performed in betweendescribed operations, described operations may be adjusted so that theyoccur at slightly different times or the described operations may bedistributed in a system which allows the occurrence of the processingoperations at various intervals associated with the processing.

In various embodiments, one or more portions of the methods andmechanisms described herein may form part of a cloud-computingenvironment. In such embodiments, resources may be provided over theInternet as services according to one or more various models. Suchmodels may include Infrastructure as a Service (IaaS), Platform as aService (PaaS), and Software as a Service (SaaS). In IaaS, computerinfrastructure is delivered as a service. In such a case, the computingequipment is generally owned and operated by the service provider. Inthe PaaS model, software tools and underlying equipment used bydevelopers to develop software solutions may be provided as a serviceand hosted by the service provider. SaaS typically includes a serviceprovider licensing software as a service on demand. The service providermay host the software, or may deploy the software to a customer for agiven period of time. Numerous combinations of the above models arepossible and are contemplated.

Various units, circuits, or other components may be described or claimedas “configured to” perform a task or tasks. In such contexts, the phrase“configured to” is used to connote structure by indicating that theunits/circuits/components include structure (e.g., circuitry) thatperforms the task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. 112, sixth paragraph, for that unit/circuit/component.Additionally, “configured to” can include generic structure (e.g.,generic circuitry) that is manipulated by software and/or firmware(e.g., an FPGA or a general-purpose processor executing software) tooperate in manner that is capable of performing the task(s) at issue.“Configured to” may also include adapting a manufacturing process (e.g.,a semiconductor fabrication facility) to fabricate devices (e.g.,integrated circuits) that are adapted to implement or perform one ormore tasks.

The foregoing description, for the purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the embodiments and its practical applications, to therebyenable others skilled in the art to best utilize the embodiments andvarious modifications as may be suited to the particular usecontemplated. Accordingly, the present embodiments are to be consideredas illustrative and not restrictive, and the invention is not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

What is claimed is:
 1. A method for cluster data replication, performedby controllers of a cluster of controllers, comprising: electing one ofthe controllers of the cluster as a master controller with the othercontrollers of the cluster being follower controllers; updating by theelected one of the controllers, in response to being elected the mastercontroller, state information in a system database of the electedcontroller to indicate that the elected one of the controllers is themaster controller; coordinating, by one or more objects in the mastercontroller that are enabled in reaction to the state information,replication of changes to the data, the system database, and the stateinformation from the master controller to the follower controllers; anddisabling, by one or more objects in each of the follower controllers inreaction to the state information, initiation of the replication ofchanges to the data, the system database and the state information byone or more objects in each of the follower controllers.
 2. The methodof claim 1, wherein the elected one of the controllers is a new mastercontroller elected from among the follower controllers in the clusterand is elected in response to a temporary dropout of an existing mastercontroller in the cluster.
 3. The method of claim 1, wherein the electedone of the controllers is a new master controller elected from among thefollower controllers in the cluster and is elected in response to afailure of an existing master controller in the cluster.
 4. The methodof claim 1, wherein each controller of the cluster further executes oneor more service processes, each service process able to set up one ormore objects that react to the state information and coordinatereplication of changes to the data, the system database and the stateinformation from a master controller in the cluster to followercontrollers in the cluster, wherein the one or more objects on each ofthe follower controllers is supportive of the follower controllersreceiving the changes from the master controller but is disabled frominitiating the replication of changes.
 5. The method of claim 4, whereineach service process has access to a data replication library from whichthe service process instantiates the one or more objects.
 6. The methodof claim 1, wherein one of the one or more objects comprises a periodicreplication object to receive one or more notifications regardingchanges in the data and to issue a synchronize request responsive toreceiving the one or more notifications and expiration of a timeinterval.
 7. The method of claim 1, wherein one of the one or moreobjects comprises a responsive replication object to receivenotifications regarding changes in the data and to issue a synchronizerequest responsive to receiving such a notification.
 8. The method ofclaim 1, wherein the state information comprises indications of whichcontrollers are members of the cluster and, for each controller that isa member of the cluster, an indication of whether the controller is amaster or a follower.
 9. The method of claim 1, wherein to set up anobject, one of the one or more service processes registers data ofinterest to the service process so that a notification is sent to theobject regarding changes in the data of interest.
 10. The method ofclaim 1, wherein in response to the state information indicating the oneor more service processes are on the master controller, and in responseto a further controller joining the cluster, the one or more serviceprocesses of the master controller update the state information in thesystem database of the master controller to indicate the furthercontroller joining membership in the cluster as a follower, wherein thestate information is propagated from the master controller to all othercontrollers in the cluster.
 11. The method of claim 1, wherein themaster controller further comprises one or more applications that areable to configure and manage a network formed by a plurality of networkelements.
 12. The method of claim 11, wherein each of the plurality ofnetwork elements includes a network element database and the mastercontroller utilizes the system database to maintain coherency of thenetwork element databases of the plurality of network elements.
 13. Themethod of claim 1, wherein the data includes a filesystem containing aplurality of files.
 14. A method for cluster data replication by acontroller in a cluster of controllers including a master controller,the method comprising: detecting that the master controller has droppedout of the cluster; updating state information in a system database ofthe controller to indicate that the controller is the new mastercontroller of the cluster; and coordinating replication of changes tothe data, the system database and the state information as the newmaster controller to any follower controller in the cluster ofcontrollers.
 15. The method of claim 14, wherein detecting that themaster controller has dropped out of the cluster comprises detecting afailure of the master controller.
 16. The method of claim 14, whereinthe cluster of controllers includes at least one follower controller andthe method further comprises the step of: preventing the at least onefollower controller from initiating changes to the data, the systemdatabase, and the state information.
 17. A cluster data replicationsystem, comprising: a plurality of network elements; and a plurality ofcontrollers coupled over a network to the plurality of network elements,the plurality of controllers forming a cluster of controllers that isable to elect one of the controllers of the cluster as a mastercontroller with the other controllers of the cluster being followercontrollers, the elected one of the controllers able to update, inresponse to being elected the master controller, state information in asystem database of the elected controller to indicate that the electedone of the controllers is the master controller, and the mastercontroller including one or more objects that are enabled in reaction tothe state information and are able to coordinate replication of changesto the data, the system database, and the state information from themaster controller to the follower controllers, and each of the followercontrollers including one or more objects able to, in reaction to thestate information, disable initiation of the replication of changes tothe data, the system database and the state information by the one ormore objects in each of the follower controllers.
 18. The cluster datareplication system of claim 17, wherein each controller of the clusterfurther includes one or more service modules, each service module ableto execute a service process to set up one or more objects that react tothe state information and coordinate replication of changes to the data,the system database and the state information from a master controllerin the cluster to follower controllers in the cluster, wherein the oneor more objects on each of the follower controllers is supportive of thefollower controllers receiving the changes from the master controllerbut is disabled from initiating the replication of changes.
 19. Thecluster data replication system of claim 18, wherein each service modulehas access to a data replication library from which the service moduleinstantiates the one or more objects.
 20. The cluster data replicationsystem of claim 17, wherein the master controller is a current mastercontroller and wherein the plurality of controllers are able to elect anew one of the plurality of controllers as the master controller inresponse to detecting a failure of the current master controller. 21.The cluster data replication system of claim 17, wherein each of theplurality of network elements comprises a virtual network element.